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ABSTRACT 

This paper presents a review of the Virtual Factory Teaching 
(VFTS) project, a Web-based, multimedia collaborative learning network. The 
system allows students, working alone or in teams, to build factories, 
forecast demand for products, plan production, establish release rules for 
new work into the factory, and set scheduling rules for workstations. 

Included in the paper are a system description and list of project tasks. The 
evaluation component involving 3 different campuses is described and the 
results of 3-years of analyses are presented, including demographic 
descriptions, self-assessment results, performance results, attitudinal 
responses, and usability. In addition, instructor observations and course 
project output is also examined as components of the VFTS evaluation effort. 
It is believed through the analysis that the VFTS is an excellent 
instructional method to teach students the integration of the different 
modules in operations planning. (Author) 
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Abstract: This paper presents a review of the Virtual Factory Teaching (VFTS) project, a 
web-based, multi-media collaborative learning network. The system allows students, 
working alone or in teams, to build factories, forecast demand for products, plan 
production, establish release rules for new work into the factory, and set scheduling rules 
for workstations. Included in the paper are a system description and list of project tasks. 
The evaluation component involving three different campuses is described and the results 
of three-years of analyses are presented, including demographic descriptions, self- 
assessment results, performance results, attitudinal responses, and usability. In addition, 
instructor observations and course project output is also examined as components of the 
VFTS evaluation effort. It is believed through the analysis that the VFTS is an excellent 
instructional method to teach students the integration of the different modules in operations 
planning. 
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Background 

To address the manufacturing educational needs of new engineers, a web-based, multi-media 
collaborative learning network, referred to as a Virtual Factory Teaching System (VFTS), was developed 
under an National Science Foundation (NSF) grant (Dessouky, et, al 1998; Dessouky, et. al, 2001; 
Kazlauskas, et. al. 2000; Kazlauskas, 2001). This tool is currently being used by engineering students from 
the University of Southern California, San Jose State University, the University of Virginia, and most 
recently North Carolina Agricultural and Technical State University. The overall aims of the project are: 

• To provide, disseminate, and evaluate a manufacturing education pedagogical tool that 
promotes student understanding of complex factory dynamics. 

• To improve student skills in communication, persuasion, negotiation, and management, as 
well as in the technical arenas of production planning, forecasting, simulation, scheduling, 
and integration. 

• To provide a forum in which engineering and business school students can participate in 
virtual teams that cut across universities. 

The research plan of the project is aimed at exploring the interface between virtual factories, 
engineering education, intelligent agents, and the Internet for new ways of teaching modem manufacturing 
problems, practices, theory, and techniques to engineering and business undergraduate students. In 
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addition, it is aimed at examining its potential for use as an information vehicle on the topic of 
manufacturing for K-12 students. Various research questions are addressed, such as: how students perform 
when using new technology-enhanced modes of learning; what are the effects on attitudes; how intelligent 
agents might assume tutoring and participative roles; and how team performance, hampered by 
geographical separation, might be enhanced via advanced communication technologies. 

The project timetable consisted of the following: a Baseline period to provide an understanding of 
the Industrial Engineering courses (with scheduling content) using traditional instructional methodologies, 
i,e, without the VFTS; the use of the VFTS to provide an understanding of the Industrial Engineering 
courses (with scheduling content) offered at the various institutions; then the use of an expanded version of 
the VFTS at the same institutions with pedagogical agents and use of virtual teaming included. The 
following presents an overview of the features of the VFTS; a description of the evaluation approaches; and 
three-year project results. 



System Description 

The architecture of the VFTS was kept simple and modular. There are three layers in the design: 
AweSim Server, VFTS Java Server and Clients, Clients, which are students in our case, use standard 
WWW browsers like Microsoft Internet Explorer, to connect to the VFTS Java Server using its Web Page, 
Most of the communication between the clients and the server takes place using Java applets. The Java- 
Server functions as a mediator between the AweSim factory servers and the clients. The Awesim Server is 
responsible for factory knowledge and simulation. The layers interface using a message protocol set up to 
minimize bandwidth requirements. Other components of the VFTS include the use of Ptolemy, a graphing 
package, and Drasys, an Operations Research (OR) package. 

The system allows students, working alone or in teams, to build factories, forecast demand for 
products, plan production, establish release rules for new work into the factory, and set scheduling rules for 
workstations. They can run simulations where an animated panel displays jobs progressing through their 
factory, with queue counts, finished goods counts, graphs, and reporting functions all available. Students 
access via the VFTS, using computers in university computer labs, or in their dorms or homes, a virtual 
representation of the factory. The professor posts assignments related to this factory over the Internet, 
“unfreezing” parameters as necessary so students may experiment without redefining the entire factory. 
Students observe the effects of their decisions, and student teams assume factory roles to solve problems; if 
they reach an impasse, intelligent agents provide guidance. Selective information may be given based on 
student roles; for example, the production supervisor may have equipment information, the engineer new 
technology information, etc. Students sort out strategies and can discuss options via e-mail and electronic 
chat rooms. Since this course is a common one found in many universities, collaboration among 
universities is feasible. Faculty members virtually “team teach” the course. Intelligent agents are 
incorporated into the VFTS to monitor student progress and provide immediate feedback. 

The latest version of the VFTS software includes support for user account management, adding 
security to allow students within a group to share data while preventing access to it from students outside 
the group, a more easy to use interface, instrumentation so that the software will gather data on student 
usage patterns, on-line help, on-line documentation and an on-line tutorial to help for students and faculty 
learn how to use the VFTS, and an introductory homework exercise to help students learn how to integrate 
the VFTS into the course, A project that uses the VFTS was developed to complement the students' 
classroom learning and was integrated into the courses at the participating universities. The latest version 
also includes a pedagogical agent that monitors students' use of the VFTS and provide guidance. This 
required integrating the pedagogical agent software (ALI) into the VFTS, adding the ability for ALI to 
maintain a persistent model of each student's knowledge across sessions, extending AU to include more 
sophisticated explanation capabilities, adding VFTS-specific knowledge to allow ALI to understand the 
instructional objectives of the VFTS and provide appropriate support to students, and instrumenting ALI to 
maintain a log of interactions with students to aid our evaluation. 

The current version VFTS is available at http://vfts,isi,cdu . 
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Project Tasks 



The VFTS project included a large set of tasks involving design, development, usage, evaluation, 
and dissemination. The various tasks associated with the VFTS project are as follows: 

■ Define instructional objectives and complete evaluation design. 

■ Develop and/or acquire evaluation instruments. 

■ Solicit feedback on the instructional objectives, evaluation design, and VFTS use from each 
university. 

■ Gather evaluation data for control group (engineering classes without the VFTS). 

■ Analyze evaluation results for control groups. 

■ Complete instrumentation of the software to support evaluation 

■ Teach engineering classes with the VFTS and gather evaluation data. 

■ Analyze evaluation results for experimental group and compare to control group. 

■ Use the evaluation results to revise and refine the VFTS and its use in the engineering 
curricula, its instrumentation, and the pedagogical agents. 

■ Teach engineering classes with the VFTS, including the use of virtual teams that span 
multiple universities, and gather evaluation data for this final experimental group. 

■ Analyze summative evaluation results. 

■ Make final revisions to the VFTS software based on evaluation results. 



Evaluation 

In the preceding list of project tasks, it should be noted that considerable emphasis is placed on the 
design and implementation of an extensive, multi-university evaluation, which is a central component of 
the VFTS project. The initial evaluation efforts were used a) to check to make sure each project step was 
implemented according to plan and that milestones are being met; and b) to consider project modification. 
This evaluation, formative in nature, assisted in modifying the design and development of the evaluation, as 
well as of the overall design of the web-based VFTS. For example, a Computer Competencies Multiple 
Domains instrument that was used initially to determine the level of entering competency in such areas as 
spreadsheets and statistical packages was dropped. It was determine through initial surveys that the learners 
were homogeneous in terms of a high level of computer competencies. Through the evaluation process, it 
was decided to gather more data on such issues on the ease of use and operation of the web-based VFTS, 
and to modify the actual design by changing the opening screen, and by including more tutorial help. 

An emphasis of this project is to evaluate learner outcomes, attitudes, and learning. This effort 
includes the use of various instruments and examination of student work. 

The following are the instruments that are presently being used in evaluation: 

• Student Pre -course and Post-course Survey instruments which are used to gather data on 
student demographics and student self-rated assessment of entering/post-course knowledge of 
operations scheduling, such as the skills, knowledge, and abilities in the use of gantt charts 
and regression and time series models . 

• Operations Scheduling Pre-test/Post-test instruments which are used measure course content 
before and after instruction and use of the VFTS. 

• Affective-Level/Attitudes Instrument (VFTS Participant Opinion Survey) Pre-test/Post-test 
instrument which are used to measure reactions to the use of the VFTS, such as reactions to 
use of simulations, working collaboratively, and relevance to future career. 

• Usability Evaluation instrument which is used to gather data on such issues on the ease of use 
and operation of the web-based VFTS. 



The VFTS Studies 

Various studies were conducted in the VFTS Project. The first of these was the Baseline 
evaluation conducted at the various universities, to provide an understanding of the Industrial Engineering 



courses (with scheduling content) using traditional instructional methodologies. The second of these was an 
investigation of the use of the VFTS at the same universities, to provide an understanding of the Industrial 
Engineering courses (with scheduling content). A second usage of the VFTS occurred, an enhanced version 
with teaming and pedagogical agency. The following represents the findings from these investigations (at 
three universities to-date). 



Demographics 

For the most part the demographic analyses showed similarities between both the Baseline and 
VFTS groups at the various universities, and the demographics appear to parallel those that are found in 
most engineering programs. Most students were in the 21-22 age category, with the age distributions at 
most universities typical of a ‘traditional’ undergraduate student. However, students at one institution had 
an older mean and the age range was quite large. At all institutions, students were mostly seniors. The 
majority of students at all three campuses were male, as reflects the common engineering enrollment 
pattern. There was no significant difference on the gender variable between the groups. The primary 
language was English but many other languages were represented, with 50% or more students at two 
institutions having a language other than English as their first language. 



Self-Assessment 

Instruments gathered pre and post data on a student’s interest in the course and on their self- 
assessment of knowledge of course content, such as whether they could apply forecasting methods to new 
problems. 

Students have a moderate interest in the course. One can support the general statement that 
students know little, if anything, about the content of the course but think that it will help them in a more 
general way. At two institutions, students indicated that they did see the course assisting in developing 
industrial engineering fundamentals. Pre and post-test measures of self-assessment indicate significant 
gains in most areas. In particular, the overall gains in self-assessment in the second year offerings of the 
VFTS at two universities were significant. The self-report assessment provides evidence that the students 
feel as if they have learned the subject in the course. 



Performance 

As a general statement, it seemed desirable to determine if there are significant differences 
between and among the participant groups. Representative analyses reflect: performance at each institution 
in both the Baseline and VFTS options; and Baseline to VFTS Comparisons. In examining Baseline and 
VFTS groups at the various institutions, analysis indicated overall gains in learning in each offering of the 
course between pre-test and post-test, and in specific content areas of scheduling. There appears to be a 
significant difference among the schools on the pretest measure, with one institution scoring significantly 
higher than the others. Comparison between the Baseline and VFTS performance at each institution 
indicated a difference on selected pre-test items. Correlations were calculated for the pretest instruments. 
Baseline to VFTS groups. Although there were some correlations between age and certain items and among 
certain items, it appears that these results were of no significance in this particular study. There were no 
significant difference between the Baseline and the VFTS groups on the final exam except for a few 
selected items. As noted, the VFTS groups did not perform as well on the pretest as the Baseline groups, 
yet scored approximately the same on the final examination. This could imply that the less strong content- 
knowledge groups, using the VFTS, performed equivalent to the stronger content-knowledge groups. The 
post-test scores at one institution are significantly higher than at the other universities. We remark that at 
only this university was the post-test a component of the student’s final grade. In fact, it was treated as the 
final examination so in this case the students took it much more seriously where they most likely studied 
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for the examination. A comparison of the post-test scores between the baseline and VFTS groups at this 
university show a significant gain for the VFTS group, although for a smaller class size. 



Attitudes 

Instruments gathered pre and post data on a student's opinions, on such scales as usefulness, 
interest, importance, difficulty, and confidence, for example whether students thought learning to use 
computer simulations to work on class assignments would be useful to their future career. Results of the 
analysis indicated that there was no significant change in the attitudes of the students, even with the use of 
the VFTS. 



Usability 

In regard to the use of the VFTS, most students seem to be using Internet Explorer on a Windows 
98 platform. Student comments on the usability of the software isolated several problems that were 
resolved in the updated version of the VFTS. The various modules well- received; for the most part the 
online documentation and tutorials not used or students commented that they were of limited value; 
students were not positive towards the use of the VFTS teaming aspect of the course. However, effort is 
being made to modify the teaming aspect of the course with the assertion that this may impact usability 
findings on this topic. 



Other Results 

Input was solicited from the instructors of the courses at the various universities. Their comments 
regarding the use of the VFTS were positive. They noted, in particular, that students using the VFTS had a 
more realistic course project than their counterparts in the Baseline classes. 

The NSF funding for this project has supported five graduate students at the University of 
Southern California. These students have gained valuable experience in Web-based software development, 
user interface principles, pedagogical agent technology, educational materials development, and 
educational evaluation. Perhaps the most valuable training for all the students is the close multidisciplinary 
collaboration between computer scientists, engineering educators, and educational researchers and 
educators. These students will have a much broader perspective on research than typical graduate students 
that work closely with others only in their own field. 



Conclusion 



The VFTS was instrumental in facilitating students’ understanding of the integration between the 
models for forecasting, production planning, material planning, inventory planning, and scheduling. 

It was demonstrated through an analysis of project reports that students had a better understanding 
of the integration between the different modules. This was illustrated by the fact that students clearly spent 
a good part of their time, with the VFTS, in testing many different scenarios, and students were able to see 
the impact of changing a parameter on the output. It is suggested that another major benefit of the VFTS is 
that a more complex system can be modeled than otherwise. Thus, it was seen that students had a more 
realistic project than their counterparts in the Baseline groups. To summarize, it is believed through the 
analysis that the VFTS is an excellent instructional method to teach students the integration of the different 
modules in operations planning. 
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